\documentclass{article}
\usepackage{fontspec}

\begin{document}
\fontsize{12}{24}\selectfont

small font - small circ and bigcirc $\dagger \circ \bigcirc$ in math environment.

small font - small circ and bigcirc \dagger \circ \bigcirc not in math environment.

{ \fontsize{48}{48}\selectfont now huge font

\begin{tabular}{l l}R1C1\dagger & R1C2\circ\\R2C1\bigcirc & R2C2\\\end{tabular}% tabular inside a group {} - If you comment this line, the document compiles with lualatex without the undesired behavior. I don't know why this situation.
}

{ \fontsize{32}{32}\selectfont now not-so-huge font

\begin{tabular}{l l}R1C1\dagger & R1C2\circ\\R2C1\bigcirc & R2C2\\\end{tabular}% tabular inside a group {} - If you comment this line, the document compiles with lualatex without the undesired behavior. I don't know why this situation.
}

small font again, but \dagger \circ \bigcirc dagger and bigcirc huge!

\dagger \circ \bigcirc both symbols again huge - I don't know why

but if you put them inside math environment, $\dagger \circ \bigcirc$ they aren't huge. %These symbols are from math environment, so the best is putting them between dollars. LuaLaTeX does not complains but XeLaTeX does... if you replace all \bigcirc in your testBad.tex you can compile with XeLaTeX without complains and with LuaLaTeX without huge bigcirc!

\end{document}