The CallBack is an internal mechanism that only allows communication with the server-side DayPilot backend. It exchanges messages in JSON format - in other words, you can't use it to open a modal dialog on the client side.
You should use TimeRangeSelectedHandling = TimeRangeSelectedHandlingType.JavaScript and a custom JavaScript code to open the modal dialog (TimeRangeSelectedJavaScript).
You can start with the example from the "Event Editing" documentation page:
https://doc.daypilot.org/scheduler/event-editing/
Just replace the "editEvent" JavaScript method with custom code that uses Boostrap popup instead.
@Html.DayPilotScheduler("dps", new DayPilotSchedulerConfig {
BackendUrl = ResolveUrl("~/Scheduler/Backend"),
...
EventClickHandling = EventClickHandlingType.JavaScript,
EventClickJavaScript = "editEvent(e.id())"
})
<script type="text/javascript">
function editEvent(id) {
// your code here
}
</script>