It is indeed a US Army Infantry officer button, but its "symmetrical" (not skewed) eagle and recessed shield date it from the 1850s through 1874. (The eagle's wings and stance were somewhat "skewed" on earlier ones, and the shield was changed to a highly-raised version in 1875.)
That version of blank-with-single-circular-line brass back was mass-produced during the civil war. When the war ended, tens-of-thousands of this button were leftovers, which meant very few needed to be manufactured, until the raised-shield design was adopted in 1875. Therefore the odds highly favor your button was manufactured during the civil war.