There is a sense in which what is most fundamental for mathematics is supposed to (a) capture primitive objects of discourse, such as integers (and a group operation such as modular addition), as well as (b) have the ability to translate the whole of the rest of mathematics (perhaps such as axiomatic set theory). In this sense, I think number theory rests on abstract algebra, even though, from a historically-inclined vantage point, it is the other way around: abstract algebra rests on number theory. I am more inclined towards the former direction, perhaps due in part to my current studies, but I could be swayed either way.
For abelian groups, the elements are indeed isomorphic to integers, for nonabelian groups, the elements can additionally be signified as shapes (like the dihedral group D3), but even these nonabelian groups can be translated into the language of numerical systems as they are, for instance, in the Sn (permutation) groups. For instance, S3 is isomorphic with D3. More generally, knowing what we know from the history and development of mathematics, it is hard to imagine a world where the language of algebra and one-dimensional lines of symbolic discourse, infused with numbers, do not take precedence over, say, continuous objects.
When looking at the fundamentality question, intuition may suggest that the continuum of real numbers should have some sort of fundamental status as objects of discourse. It may be a matter of circumstance and aesthetic choices. For example, someone who is a topologist or differential geometer may shy away from numbers altogether in favor of manifolds or some such objects, if such an inclination is even possible in the contemporary world.