Purpose: To evaluate six cerebral arterial segmentation algorithms in a set of patients with a wide range of hemodynamic characteristics to determine real-world performance.

Approach: Time-of-flight magnetic resonance angiograms were acquired from 33 subjects: normal controls (N  =  11), sickle cell disease (N  =  11), and non-sickle anemia (N  =  11) using a 3 Tesla Philips Achieva scanner. Six segmentation algorithms were tested: (1) Otsu’s method, (2) K-means, (3) region growing, (4) active contours, (5) minimum cost path, and (6) U-net machine learning. Segmentation algorithms were tested with two region-selection methods: global, which selects the entire volume; and local, which iteratively tracks the arteries. Five slices were manually segmented from each patient by two readers. Agreement between manual and automatic segmentation was measured using Matthew’s correlation coefficient (MCC).

Results: Median algorithm segmentation times ranged from 0.1 to 172.9 s for a single angiogram versus 10 h for manual segmentation. Algorithms had inferior performance to inter-observer vessel-based (p  <  0.0001, MCC  =  0.65) and voxel-based (p  <  0.0001, MCC  =  0.73) measurements. There were significant differences between algorithms (p  <  0.0001) and between patients (p  <  0.0042). Post-hoc analyses indicated (1) local minimum cost path performed best with vessel-based (p  =  0.0261, MCC  =  0.50) and voxel-based (p  =  0.0131, MCC  =  0.66) analyses; and (2) higher vessel-based performance in non-sickle anemia (p  =  0.0002) and lower voxel-based performance in sickle cell (p  =  0.0422) compared with normal controls. All reported MCCs are medians.

Conclusions: The best-performing algorithm (local minimum cost path, voxel-based) had 9.59% worse performance than inter-observer agreement but was 3 orders of magnitude faster. Automatic segmentation was non-inferior in patients with sickle cell disease and superior in non-sickle anemia.