This phase I/II trial studies the side effects and best dose of nivolumab when given with or without ipilimumab to see how well they work in treating younger patients with solid tumors or sarcomas that have come back (recurrent) or do not respond to treatment (refractory). Monoclonal antibodies, such as nivolumab and ipilimumab, may block tumor growth in different ways by targeting certain cells. It is not yet known whether nivolumab works better alone or with ipilimumab in treating patients with recurrent or refractory solid tumors or sarcomas.