Without going into specific policies that each the Republicans and Libertarians support, what are the fundamental differences between the two?
Republicans seek to have a weak federal government and a strong state government whereas the Libertarians just want all government to be weak.
At a procedural level, this means that Republicans will generally support a decision made by the state. But it is important to remember that Republicans see a clear distinction between what is the state's responsibility and the federal government's responsibility. Things like national defense, border security and trade with foreign nations are to be left to the federal government according to the Republicans. Generally, this Republican procedural philosophy is coupled with a conservative substantive philosophy as opposed to the liberal side. This is because conservatism lends itself naturally to Republicanism. A Republican system is naturally a strong federalism perspective, i.e. Republicans want a clear separation between the federal government and the state. Conservatives like this system of government because it leads to a smaller government. A strong, centralized federal government like the liberals want would be to intrusive for the conservatives. If the federal government does not meddle in state issues, it would be a lot smaller.
The Libertarians, on the other hand, see the federal government and states in the same light. They believe it is the government vs the individual rather than, as Republicans see it, the federal government vs the states.
The reason for the Republican weak federal government/strong state government is because it is easy for an individual to have a voice in their state while it is not easy for an individual to have a voice in the federal government. Any individual can be involved in their municipality or go down to the state capitol. It is much more difficult to find a voice in Washington. Because there is a closer connection between the state and the individual than is between the federal government and the individual, it is more okay for the state to do something that the Libertarians would consider to be 'encroachment' than it is for the federal government. If you are encroached and have a voice, that is less bad than being encroached upon and not having a voice.
The Libertarians do not take any of that into account. It is simply, 'don't tread on me'. No more. While that is generally a good philosophy, it is more okay for the state to do something the Libertarians don't like than it is for the federal government because at the state level a remedy is easier to obtain.