Okay never mind, I just realized that the AppointmentSchema.IsAllDayEvent was not being loaded as a part of the CalendarView PropertySet. For those having this same issue using Exchange Web Services be sure to add AppointmentSchema.IsAllDayEvent to your CalendarView PropertySet when loading your appointments:
Dim calendar As CalendarFolder = FindDefaultCalendarFolder()
Dim cView As New CalendarView(startDate, endDate, 50)
cView.PropertySet = New PropertySet(AppointmentSchema.Subject, AppointmentSchema.Start, AppointmentSchema.End, AppointmentSchema.Id, AppointmentSchema.IsAllDayEvent) '<=== Make sure you add "AppointmentSchema.IsAllDayEvent"
Dim appointments As FindItemsResults(Of Appointment) = calendar.FindAppointments(cView)