google cloud platform – Vertex AI Gemini 2.5 Flash supervised fine-tuning job stuck indefinitely in “Preparing for tuning” state
I’m running a Supervised Fine-Tuning (SFT) job on Vertex AI using gemini-2.5-flash as the base model, and the job gets stuck indefinitely in the “Preparing for tuning…” state with the message:
Tuning metrics are being generated. Metrics will appear here when ready.
The job status shows “Running,” but it never progresses past this stage, and no metrics or errors are ever generated.
What I’ve tried / ruled out:
- Tested with multiple dataset sizes: a full dataset (~300 examples, ~1.6M total tokens) and a trivial test dataset (5 examples, ~31K tokens). Both show the identical stall — the small dataset also sat in “Preparing” for 45+ minutes with zero progress, which rules out dataset size or token volume as the cause.
- Tested with different epoch counts (2, 4, and 32) — no difference in behavior.
- Tested in two different regions (
us-central1andus-south1) — same stall in both. - Confirmed via Cloud Console that “Global concurrent tuning jobs” quota is not the bottleneck (usage correctly shows 1/1 allocated to the single active job, no contention).
- The job’s audit log (Cloud Logging) shows a successful
CreateTuningJobcall withstatus: {0}(OK) and the job enteringJOB_STATE_PENDING, but no further log entries appear afterward, even after several hours. - Attempting to launch in a third region (
northamerica-northeast1) immediately failed withINVALID_ARGUMENT: Base model gemini-2.5-flash is not supported, suggesting regional support for this model’s tuning is inconsistent — though this doesn’t explain why the supported regions also stall.
Separately noticed (possibly unrelated): the Tuning Job “Details” tab in the console shows Tuning method: Multi-Step Reinforcement Learning, despite explicitly selecting “Supervised fine-tuning” at job creation. However, the main Tuning jobs list view correctly shows Method: Supervised for the same job — so this looks like a cosmetic display bug on the Details tab rather than the backend actually running a different tuning method.
Question: Has anyone else run into Gemini 2.5 Flash SFT jobs stalling permanently in the “Preparing for tuning” stage? Is this a known regional capacity/provisioning issue, or is there a configuration step I might be missing that isn’t obvious from the docs?
Any pointers — including how to get better visibility into what’s happening during this “preparation” phase via Cloud Logging or the API — would be appreciated.
Read more here: Source link
