A conservative majority on the Supreme Court has signaled skepticism toward executive branch enforcement mechanisms, indicating a ruling that could restrict the ability of federal agencies to levy fines without jury trials.