There are fairly advanced plans to provide a hourly service Dublin Belfast in a similar fashion to Dublin Cork, its currently at the funding stage once that is in place it should result in a significant improvement
As things stand currently if there was an earlier service it would have to leave Belfast at about 4:50 am, arriving Dublin about 6:55 am which isn't terribly practical
As James has pointed out it simply isn't possible under current safety rules to operate a IE commuter train within Northern Ireland. Irish Rail has fairly generous rights to operate trains cross border as do NIR provided the trains carry the correct safety equipment, the Irish equipment being substantially more advanced than the NIR equipment
However if you have several million to spend you can set up your own rail company and leverage the EU cross border open access rules to run the service yourself, if you need a general manager I'm available
