Any economic transaction between a director and the organization creates a conflict of interest, but that doesn’t mean that you can’t do the transaction.  It should be fully disclosed and approved by directors who are not involved and who assure themselves that it is fair and reasonable for the organization.  It is a time-honored tradition for directors and other supporters to lend nonprofits money to tide them over rough spots. They can often keep organizations afloat that would otherwise sink from their financial difficulties.
I would have some concern if the number of lenders is large and the amounts are substantial enough to be personally significant.  If you have difficulty repaying the loans when due and if the lenders have not been prepared from the outset to write them off if necessary, you could have a significant block of your directors looking at your activities through a lens of trying to protect their economic self-interest.  That is undoubtedly not the best point of view for your organization.
(I should point out that the foregoing discussion does not apply to a private foundation.  Loans between private foundations and directors (other than an interest free loan from a director to the foundation) are self-dealing transactions and subject to big taxes. They should not be done.)