In case of large datasets - yes, because it will cause using temporary + using filesort, which is not good for performance and memory.
Small datasets (like “select * from table where (some condition giving 10 rows) order by rand()”) is ok. Anyway, do not forget to use EXPLAIN for your queries (query planner drives me crazy sometimes), and don’t hesitate to use workarounds in cases when your DB is doing heavy non-DB job.