Aims. We carried out optical polarimetry of five dense cores, (IRAM 04191, L1521F, L328, L673-7, and L1014) which are found to harbour very low luminosity objects (VeLLOs; Lint ≲ 0.1 L⊙). This study was conducted mainly to understand the role played by the magnetic field in the formation of very low and substellar mass range objects. Methods. Light from the stars, while passing through the dust grains that are aligned with their short axis parallel to an external magnetic field, becomes linearly polarised. The polarisation position angles measured for the stars can provide the plane-of-the sky magnetic field orientation. Because the light in the optical wavelength range is most efficiently polarised by the dust grains typically found at the outer layers of the molecular clouds, optical polarimetry mostly traces the magnetic field orientation of the core envelope. Results. The polarisation observations of stars projected on IRAM 04191, L328, L673-7, and L1014 were obtained in the R-band and those of L1521F were obtained in the V-band. The angular offsets between the envelope magnetic field direction (inferred from optical polarisation measurements) and the outflow position angles from the VeLLOs in IRAM 04191, L1521F, L328, L673-7, and L1014 are found to be 84°, 53°, 24°, 08°, and 15°, respectively. The mean value of the offsets for all the five clouds is ~ 37°. If we exclude IRAM 04191, the mean value reduces to become ~ 25°. In IRAM 04191, the offset between the projected envelope and the inner magnetic field (inferred from the submillimetre data obtained using SCUBA-POL) is found to be ~ 68°. The inner magnetic field, however, is found to be nearly aligned with the projected position angles of the minor axis, the rotation axis of the cloud, and the outflow from the IRAM 04191-IRS. We discuss a possible explanation for the nearly perpendicular orientation between the envelope and core scale magnetic fields in IRAM 04191. The angular offset between the envelope magnetic field direction and the minor axis of IRAM 04191, L1521F, L673-7, and L1014 are 82°, 60°, 47°, and 55°, respectively. The mean value of the offsets between the envelope magnetic field and the minor axis position angles for the four cores is found to be ~ 60°. Conclusions. The results obtained from our study on the limited sample of five cores with VeLLOs show that the outflows in three of them tend to nearly align with the envelope magnetic field.